Storage medium storing virtual position determining program

ABSTRACT

A game controller includes an image-capturing section, and outputs, as control data, a captured image itself or a result of performing a predetermined operation on the captured image. Using the control data obtained from the game controller, distance data regarding the distance between the game controller and the imaging target is calculated. A predetermined game process is performed on a virtual game world based on the calculated distance data. Then, the virtual game world after the game process is displayed as an image on a display device.

CROSS REFERENCE TO RELATED APPLICATION

The disclosure of Japanese Patent Application No. 2005-267717 isincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a storage medium storing a virtualposition determining program and, more particularly, to a storage mediumstoring a virtual position determining program for calculating theposition of a virtual object in a virtual space based on the position ofa target in the real space.

2. Description of the Background Art

In the prior art, Japanese Laid-Open Patent Publication No. 11-305935(hereinafter “Patent Document 1”), for example, discloses a positiondetection system for taking an image of an infrared light source andanalyzing the obtained image to determine a gunsight position.

As an example of a position detection system, Patent Document 1discloses a shooter video game in which the player shoots targets byusing, as a game controller, a gun-shaped controller provided with animage capturing device. Four infrared light sources are provided, asimaging targets, each at a corner of a cathode-ray tube, and an imagethereof is taken by the image capturing device. The position on thecathode-ray tube being pointed at by the gun-shaped controller isdetected based on the positions of the imaging targets on the obtainedimage. A virtual shooter game is played by using the position beingpointed at by the gun-shaped controller as the gunsight position. Asstated in Paragraph 21 of Patent Document 1, in a case where thedistance between the image capturing device and the target (marker) canbe changed freely, the gunsight position can be detected accurately ifthe image capturing device has a controllable zoom function and if thezoom function is controlled so that the marker on the obtained imagealways has an optimal size.

However, there is no such concept in the prior art that the position ofthe game controller, being an example of an input device, in the realspace is used as a control input. In the position detection systemdisclosed in Patent Document 1, the position on the cathode-ray tubebeing pointed at by the gun-shaped controller is used as an input, butthe pointed position only represents a position on the screen, not theposition of the controller in the real space. While the distance betweenthe image capturing device and the marker in the real space can beobtained, the distance is only used for making corrections. In otherwords, the distance is a sort of a value representing the position ofthe controller in the real space, but it is not used as a control input.

If the distance between the game controller and the imaging target isused as a control input, the range over which the player can move thegame controller in the distance direction will be quite large.Therefore, the input range and the input variation will be significant,and it is difficult in the prior art to appropriately associate such avalue with significant variations with the position of the controlledobject in the virtual space. Moreover, where the player makes thecontrol input varies depending on the situation, thereby varying thedistance value, and it is difficult to associate it with an appropriatevirtual position. It is clear that similar problems will occur not onlywith the distance, which is a one-dimensional value, but also whendetecting a three-dimensional position in the real space. With thoseproblems as set forth above, there is no such concept in conventionalcontrol input methods that the position of the controller in the realspace is used directly as a control input.

SUMMARY OF THE INVENTION

Therefore, an object of the present invention is to provide a storagemedium storing a virtual position determining program, which realizes anovel control operation with a high degree of freedom while using theposition of a target in the real space.

The present invention has the following features to attain the objectsmentioned above.

A first aspect of the present invention is directed to a storage mediumstoring a virtual position determining program for instructing acomputer (30) to perform a process on a virtual space based on inputinformation (Da) obtained from an input device (7), the inputinformation including target data (Da1, Da2) that changes according to achange in a position of a predetermined target in a real space, theprogram instructing the computer to perform: an actual position datacalculation step (S52) of calculating actual position data (Db3)representing a position of the target in the real space by using thetarget data; an effective range defining step (S63, S66) of defining aneffective range as a range extending from a first boundary to a secondboundary, wherein the first boundary is represented by first positiondata and the second boundary is represented by second position data; aneffective range position data calculation step (S95, S105, S130) ofcalculating effective range position data representing a position of avalue of the actual position data in the effective range; a positiondetermining step (S67, S54) of determining a position of a controlledobject in the virtual space through an operation based on a value of theeffective range position data; and a display process step (S56) ofdisplaying the virtual space on a display device (2).

According to a second aspect, in the first aspect, the virtual positiondetermining program instructs the computer to further perform aneffective range updating step (S91 to S94, S101 to S104, S121 to S129)of, when the value of the actual position data is outside the effectiverange, updating the effective range by moving the first boundary and thesecond boundary so that the value of the actual position data isincluded in the effective range.

According to a third aspect, in the first aspect, the virtual positiondetermining program instructs the computer to further perform aneffective range updating step (S91 to S94) of, when the value of theactual position data is outside the effective range, updating theeffective range by moving the first boundary and the second boundary sothat one of the first boundary and the second boundary that is closer tothe value of the actual position data becomes equal to the value of theactual position data.

According to a fourth aspect, in the first aspect, the virtual positiondetermining program instructs the computer to further perform a secondeffective range updating step (S121 to S129) of, when the value of theactual position data is outside the effective range, updating theeffective range by moving the first boundary and the second boundary sothat one of the first boundary and the second boundary that is closer tothe value of the actual position data approaches the value of the actualposition data at a predetermined rate.

According to a fifth aspect, in the first aspect, the effective rangeposition data calculation step calculates the effective range positiondata so that the effective range position data takes a maximum valuewhen the value of the actual position data is equal to the firstboundary and the effective range position data takes a minimum valuewhen the value of the actual position data is equal to the secondboundary.

According to a sixth aspect, in the first aspect, the effective rangedefining step includes a step of further defining a play range having apredetermined length at a front end of the effective range and a rearend thereof; and the virtual position determining program instructs thecomputer to further perform an effective range updating step (S101 toS104) of, when the actual position data takes a value outside a combinedrange including the effective range and the play ranges, updating theeffective range by moving the values of the first boundary and thesecond boundary so that one of the first boundary and the secondboundary that is closer to the value of the actual position data becomesequal to the value of the actual position data.

According to a seventh aspect, in the sixth aspect, the effective rangeposition data calculation step includes a step (S106) of calculating theeffective range position data so that the effective range position datatakes a maximum value when the value of the actual position data isequal to the first position data or exceeds the effective range in adirection of the first position data, and the effective range positiondata takes a minimum value when the value of the actual position data isequal to the second position data or exceeds the effective range in adirection of the second position data.

According to an eighth aspect, in the first aspect, the effective rangedefining step defines the second position data on a first side of thefirst position data, third position data on a second side of the firstposition data, and fourth position data on the first side of the secondposition data; and the effective range defining step defines an upperlimit boundary as position data variable from the first position data tothe third position data and a lower limit boundary as position datavariable from the second position data to the fourth position data, anddefining the effective range as a range between the variable upper limitboundary and the variable lower limit boundary; and the virtual positiondetermining program instructs the computer to further perform: a stepof, when the value of the actual position data is within the variablerange of the upper limit boundary/lower limit boundary, updating theupper limit boundary/lower limit boundary so that the upper limitboundary/lower limit boundary approaches the value of the actualposition data at a predetermined rate, and updating the lower limitboundary/upper limit boundary so that the lower limit boundary/upperlimit boundary approaches the second position data/first position dataat a predetermined rate; a step of, when the value of the actualposition data is within a range between the first position data and thesecond position data, updating the upper limit boundary/lower limitboundary so that the upper limit boundary/lower limit boundaryapproaches the first position data/second position data at apredetermined rate; a step of, when the value of the actual positiondata is further on the second side of the third position data, movingthe first position data, the second position data, the third positiondata, the fourth position data and the lower limit boundary so that avalue of the third position data approaches the value of the actualposition data at a predetermined rate and updating the upper limitboundary to the new second position data; and a step of, when the valueof the actual position data is further on the first side of the fourthposition data, moving the first position data, the second position data,the third position data, the fourth position data and the upper limitboundary so that a value of the fourth position data approaches thevalue of the actual position data at a predetermined rate and updatingthe lower limit boundary to the new fourth position data (S121 to S129).

According to a ninth aspect, in the eighth aspect, the effective rangeposition data calculation step includes a step (S106) of calculating theeffective range position data so that the effective range position datatakes a maximum value when the value of the actual position data isequal to the first position data or exceeds the effective range in adirection of the first position data, and the effective range positiondata takes a minimum value when the value of the actual position data isequal to the second position data or exceeds the effective range in adirection of the second position data.

A tenth aspect of the present invention is directed to a storage mediumstoring a virtual position determining program for, based on data (Da)from an input device (7) including position information of a target in areal space, calculating a position of the target obtained whenrepresenting the real space in a predetermined coordinate system, andcalculating a virtual position of a virtual object in a virtual spacebased on the calculated position, the virtual position determiningprogram instructs a computer (30) to function as: effective rangedefining means (S63, S66) for defining an effective range, which isdefined by at least a first position and a second position in apredetermined coordinate system of the real space; normalization means(S95, S105, S130) for, when the position of the target is included inthe effective range, normalizing the effective range and calculating avalue corresponding to the position of the target obtained when thenormalization is performed; and calculation means (S67, S54) forcalculating the virtual position based on the value calculated by thenormalization means.

According to an eleventh aspect, in the tenth aspect, the virtualposition determining program instructs the computer to further functionas effective range moving means (S91 to S94, S101 to S104, S121 to S129)for, when the position of the target is not included in the effectiverange, updating the first position and the second position to move theeffective range to a position that includes the position of the target.

According to the first aspect, an effective range of the position datais defined, effective range position data representing the position ofthe value of the actual position data in the effective range iscalculated, and the position of the controlled object in the virtualspace is determined through an operation based on the effective rangeposition data. Therefore, it is possible to realize a virtual positiondetermining process with a novel operation feel for determining avirtual position based on the position of the target in the real space.

According to the second aspect, the boundary is moved when the value ofthe actual position data is outside the effective range, whereby it ispossible to freely define the range of motion of the target, thusincreasing the degree of freedom.

According to the third aspect, the boundary is moved when the value ofthe actual position data is outside the effective range, whereby it ispossible to freely define the range of motion of the target, thusincreasing the degree of freedom.

According to the fourth aspect, the boundary is moved when the value ofthe actual position data is outside the effective range, whereby it ispossible to freely define the range of motion of the target, thusincreasing the degree of freedom.

According to the fifth aspect, the effective range position data takesthe maximum value when the value of the actual position data is equal tothe first boundary and the effective range position data takes theminimum value when the value of the actual position data is equal to thesecond boundary. Therefore, it is possible to intuitively control theposition of the controlled object in the virtual space according to theposition of the target.

According to the sixth aspect, the play range is provided at the frontend and the rear end of the effective range, whereby it is possible toreduce the influence of an abrupt movement beyond the effective rangedue to an erroneous operation, or the like.

According to the eighth aspect, since the boundary is moved, it ispossible to freely define the range of motion of the target. Moreover,the effective range is moved gradually. Therefore, it is possible toreduce the influence of an abrupt movement beyond the effective rangedue to an erroneous operation, or the like.

According to the seventh or ninth aspect, the effective range positiondata takes the maximum value when the value of the actual position datais equal to the first position data or exceeds the effective range inthe direction of the first position data, and the effective rangeposition data takes the minimum value when the value of the actualposition data is equal to the second position data or exceeds theeffective range in the direction of the second position data. Therefore,it is possible to reduce the influence of an abrupt movement beyond theeffective range due to an erroneous operation, or the like, and tointuitively control the position of the controlled object in the virtualspace according to the position of the target.

According to the tenth aspect, an effective range of the position isdefined, and the virtual position is determined through an operationbased on a value corresponding to the position of the target obtainedwhen the effective range is normalized. Therefore, it is possible torealize a virtual. position determining process with a novel operationfeel for determining a virtual position based on the position of thetarget in the real space, and to intuitively control the position of thecontrolled object in the virtual space according to the position of thetarget.

According to the eleventh aspect, the effective range is moved when theposition is outside the effective range, whereby it is possible tofreely define the range of motion of the target, thus increasing thedegree of freedom.

These and other objects, features, aspects and advantages of the presentinvention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an external view of a video game system 1 in one embodimentof the present invention;

FIG. 2 is a functional block diagram showing a video game device 3 ofFIG. 1;

FIG. 3 is a perspective view showing a controller 7 of FIG. 1 as viewedfrom the upper rear side;

FIG. 4 is a perspective view showing the controller 7 of FIG. 3 asviewed from the lower rear side.

FIG. 5A is a perspective view showing the controller 7 of FIG. 3 with anupper casing taken off;

FIG. 5B is a perspective view showing the controller 7 of FIG. 3 with alower casing taken off;

FIG. 6 is a block diagram showing a configuration of the controller 7 ofFIG. 3;

FIG. 7 generally shows how the player uses the controller 7 of FIG. 3 toperform a game operation;

FIG. 8 shows the controller 7 being held in the player's right hand, asviewed from the front side;

FIG. 9 shows the controller 7 being held in the player's right hand, asviewed from the left side;

FIG. 10 shows viewing angles between markers 8L and 8R and an imagecapturing/processing section 74;

FIG. 11 shows important data to be stored in a main memory 33 of thevideo game device 3;

FIG. 12 is a flow chart showing the game process performed by the videogame device 3;

FIG. 13 shows, in detail, a subroutine of step 52 in FIG. 12 for theprocess of calculating control status information, pointed position dataand depth position data;

FIG. 14 shows, in detail, a subroutine of steps 62 and 65 in FIG. 13 forthe process of calculating a distance d;

FIG. 15 shows, in detail, a subroutine of step 66 in FIG. 13 for theprocess of calculating the forward/backward position parameter andre-defining the effective range;

FIG. 16 shows the operation of calculating the distance d;

FIGS. 17A and 17B show the operation of calculating the forward/backwardposition parameter r and defining the effective range according to thesubroutine shown in FIG. 13;

FIG. 18 shows an exemplary relationship between the forward/backwardposition parameter r and the depth position data z defined in athree-dimensional game space;

FIG. 19 shows another exemplary relationship between theforward/backward position parameter r and the depth position data zdefined in the three-dimensional game space;

FIG. 20 shows, in detail, a subroutine for the process of calculatingthe forward/backward position parameter and re-defining the effectiverange according to a second embodiment of the present invention;

FIG. 21 shows, in detail, a subroutine of step 106 in FIG. for therounding process;

FIGS. 22A and 22B show the operation of calculating the forward/backwardposition parameter r and defining the effective range according to thesubroutine shown in FIG. 20;

FIG. 23 shows, in detail, a subroutine for the process of calculatingthe forward/backward position parameter and re-defining the effectiverange according to a third embodiment of the present invention; and

FIGS. 24A and 24B show the operation of calculating the forward/backwardposition parameter r and defining the effective range according to thesubroutine shown in FIG. 23.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

(First Embodiment)

Referring to FIG. 1, a video game system 1 for executing a virtualposition determining program according to a first embodiment of thepresent invention will now be described. FIG. 1 shows an external viewof the video game system 1. In the present embodiment, the video gamesystem 1 includes a home-console type video game device.

Referring to FIG. 1, the video game system 1 includes a home-consoletype video game device (hereinafter simply “video game device”) 3 and acontroller 7 for giving control information to the video game device 3.The video game device 3 is connected, via a connection cord, to adisplay (hereinafter “monitor”) 2 provided with a speaker 2 a, such as ahome television receiver. A receiver unit 6 is connected to the videogame device 3 via a connection terminal. The receiver unit 6 receivestransmit data wirelessly transmitted from the controller 7, and thecontroller 7 and the video game device 3 are connected via wirelesscommunications. The video game device 3 uses an optical disk 4, being anexample of an information storage medium, that can be received by thevideo game device 3. Provided on the upper principal plane of the videogame device 3 are an ON/OFF switch for turning ON/OFF the power of thevideo game device 3, a reset switch for resetting a game process, and anOPEN switch for opening the upper lid of the video game device 3. Thelid opens up when the OPEN switch is pressed by the player so that theoptical disk 4 can be put in place.

The video game device 3 can also receive an external memory card 5including a backup memory, or the like, for statically storing savedata, or the like. The video game device 3 executes a video gameprogram, or the like, stored in the optical disk 4 to obtain a gameimage, and displays the obtained game image on the monitor 2. The videogame device 3 may reproduce a past game status from save data stored inthe external memory card 5 to obtain a game image for that past gamestatus, and display the obtained game image on the monitor 2. Then, theplayer of the video game device 3 can enjoy the game process byoperating the controller 7 while watching the game image displayed onthe monitor 2.

The controller 7 wirelessly transmits, from a communications section 75(to be described later) therein, transmit data to the video game device3, to which the receiver unit 6 is connected, by means of a techniquesuch as Bluetooth (registered trademark), for example. The controller 7is means for controlling a player object in a game space displayed onthe monitor 2. The controller 7 has a control section, including aplurality of control buttons, keys, sticks, etc. As will be moreapparent from the following description, the controller 7 includes animage capturing/processing section 74 for capturing an image as viewedfrom the controller 7. As an example of imaging targets to be capturedby the image capturing/processing section 74, two LED modules(hereinafter “markers”) 8L and 8R are provided around the display screenof the monitor 2. The markers 8L and 8R output infrared light to thefront side of the monitor 2.

Referring now to FIG. 2, a configuration of the video game device 3 willbe described. FIG. 2 is a functional block diagram showing the videogame device 3.

Referring to FIG. 2, the video game device 3 includes a RISC CPU(Central Processing Unit) 30, for example, for executing variousprograms. The CPU 30 executes a boot program stored in a boot ROM (notshown), thus initializing memory devices, such as a main memory 33, andthen executes a video game program stored in the optical disk 4 toperform a game process, etc., according to the video game program.Connected to the CPU 30 via a memory controller 31 are a GPU (GraphicsProcessing Unit) 32, the main memory 33, a DSP (Digital SignalProcessor) 34, and an ARAM (Audio RAM) 35. The memory controller 31 isconnected, via a predetermined bus, to a controller I/F (interface) 36,a video I/F 37, an external memory I/F 38, an audio I/F 39 and a diskI/F 41, which are connected to the receiver unit 6, the monitor 2, theexternal memory card 5, the speaker 2 a and a disk drive 40,respectively.

The GPU 32 is responsible for image processing based on instructionsfrom the CPU 30, and is a semiconductor chip, for example, capable ofcomputations necessary for 3D graphics display. The GPU 32 performs theimage process by using a memory dedicated for image processing (notshown) or a part of the memory area of the main memory 33. The GPU 32produces game image data or movie data to be displayed on the monitor 2using these memory areas, and outputs the produced data to the monitor 2via the memory controller 31 and the video I/F 37 as necessary.

The main memory 33 is a memory area used by the CPU 30, and stores avideo game program, etc., as necessary for processes performed by theCPU 30. For example, the main memory 33 stores the video game programloaded from the optical disk 4 by the CPU 30 and various data, etc. Thevideo game program, the various data, etc., stored in the main memory 33are executed or processed by the CPU 30.

The DSP 34 is for processing sound data, etc., produced by the CPU 30when executing the video game program, and is connected to the ARAM 35for storing the sound data, etc. The ARAM 35 is used when the DSP 34performs a predetermined process (e.g., storing a video game program,sound data, etc., which have been loaded in advance). The DSP 34 readsout the sound data stored in the ARAM 35, and outputs the sound datathrough the speaker 2 a provided in the monitor 2 via the memorycontroller 31 and the audio I/F 39.

The memory controller 31 is responsible for the overall control of datatransfers, and is connected to the various I/F's described above. Thecontroller I/F 36 includes, for example, four controller I/F portions 36a to 36 d, each having a connector into which an external unit can befitted for communicable connection between the external unit and thevideo game device 3. For example, the receiver unit 6 may be fitted intothe connector to be connected to the video game device 3 via thecontroller I/F 36. As described above, the receiver unit 6 receivestransmit data from the controller 7, and outputs the transmit data tothe CPU 30 via the controller I/F 36. The monitor 2 is connected to thevideo I/F 37. The external memory card 5 is connected to the externalmemory I/F 38, whereby a backup memory, etc., provided in the externalmemory card 5 can be accessed. The audio I/F 39 is connected to thespeaker 2 a provided in the monitor 2 so that the sound data read outfrom the ARAM 35 by the DSP 34 or the sound data outputted directly fromthe disk drive 40 can be outputted through the speaker 2 a. The disk I/F41 is connected to the disk drive 40. The disk drive 40 reads out datafrom the optical disk 4 placed in a predetermined read-out position, andoutputs the data to the bus or the audio I/F 39 of the video game device3.

Referring now to FIGS. 3 and 4, the controller 7 will be described. FIG.3 is a perspective view showing the controller 7 as viewed from theupper rear side. FIG. 4 is a perspective view showing the controller 7as viewed from the lower rear side.

The controller 7 shown in FIGS. 3 and 4 includes a housing 71 formed bymolding a plastic material, for example, and a plurality of controlsections 72 are provided on the housing 71. The housing 71 has agenerally rectangular parallelepiped shape, with the longitudinaldirection being the front-rear direction, and has an overall size suchthat it can be held in a hand of an adult or a child.

A cross-shaped key 72 a is provided on the upper surface of the housing71, centered in the left/right direction and near the front end. Thecross-shaped key 72 a is a cross-shaped four-way push switch, in whichfour control portions associated with four different directions(forward, backward, left and right) indicated by arrows are provided inthe protruding portions of the cross shape while being spaced apart fromone another by 90°. The player can select one of the forward, backward,left and right directions by pressing down a corresponding one of thecontrol portions of the cross-shaped key 72 a. For example, the playercan control the cross-shaped key 72 a to move a player character, etc.,in a virtual game world, or move a cursor, in a certain direction.

While the cross-shaped key 72 a is a control section that outputs acontrol signal according to a direction input operation by the player,it may be any other suitable type of a control section. For example, thecross-shaped key 72 a may be replaced by a combination switch includinga ring-shaped four-way push switch and a center switch surrounded by thering-shaped push switch. Alternatively, the cross-shaped key 72 a may bereplaced by a stick-shaped control section protruding from the uppersurface of the housing 71, which outputs a control signal according tothe direction in which it is tilted. Alternatively, the cross-shaped key72 a maybe replaced by a horizontally-movable (slidable) disc-shapedcontrol section, which outputs a control signal according to thedirection in which it is slid. Alternatively, the cross-shaped key 72 amay be replaced by a touch pad. Alternatively, the cross-shaped key 72 amay be replaced by a control section including at least four switchesassociated with different directions (forward, backward, left andright), which outputs a control signal according to the switch pressedby the player.

A plurality of control buttons 72 b to 72 g are provided on the uppersurface of the housing 71, closer to the rear end with respect to thecross-shaped key 72 a. The control buttons 72 b to 72 g are controlsections, each of which outputs a control signal associated therewithwhen being pressed by the player. For example, the control buttons 72 bto 72 d may be assigned a function as an X button, a Y button and a Bbutton, respectively. For example, the control buttons 72 e to 72 g maybe assigned a function as a select switch, a menu switch and a startswitch, respectively. Each of the control buttons 72 b to 72 g isassigned a function as specified in the video game program executed bythe video game device 3, the details of which will not be discussedherein as being not directly related to the description of the presentinvention. In the arrangement shown in FIG. 3, the control buttons 72 bto 72 d are arranged in the forward/backward direction while beingcentered in the left/right direction on the upper surface of the housing71. The control buttons 72 e to 72 g are arranged in the left/rightdirection between the control buttons 72 b and 72 d on the upper surfaceof the housing 71. The control button 72 f is buried under the uppersurface of the housing 71 so as to prevent the player from pressing thebutton unintentionally.

A control button 72 h is provided on the upper surface of the housing71, closer to the front end with respect to the cross-shaped key 72 a.The control button 72 h is a power switch for remotely turning ON/OFFthe power of the video game device 3 from a remote position. The controlbutton 72 h is also buried under the upper surface of the housing 71 soas to prevent the player from pressing the button unintentionally.

A plurality of LEDs 702 are provided on the upper surface of the housing71, closer to the rear end with respect to the control button 72 c. Thecontroller 7 is given a controller ID (number) for identifying thecontroller 7 from others. The LEDs 702 may, for example, be used fornotifying the player of the controller ID being currently assigned tothe controller 7. Specifically, when transmit data is transmitted fromthe controller 7 to the receiver unit 6, one or more of the LEDs 34 arelit depending on the controller ID.

A depressed portion is formed on the lower surface of the housing 71. Aswill later be more apparent, the depressed portion of the lower surfaceof the housing 71 is located where the index or middle finger of theplayer lies when the player holds the controller 7. A control button 72i is provided on a slope on the rear side of the depressed portion. Forexample, the control button 72 i is a control section that functions asan A button, and used as a trigger switch in a shooter video game or fordirecting the attention of a player object toward a predeterminedobject.

An image sensing device 743, forming a part of the imagecapturing/processing section 74, is formed on the front side of thehousing 71. The image capturing/processing section 74 is a system foranalyzing image data obtained by the controller 7 to determine each spotwith high luminance and then to detect the centroid and the sizethereof, and has a maximum sampling frequency of about 200 frames persecond, for example, and is thus capable of following fast movements ofthe controller 7. The details of the configuration of the imagecapturing/processing section 74 will be described later. A connector 73is provided on the rear side of the housing 71. The connector 73 is, forexample, a 32-pin edge connector, and is used for connection between thecontroller 7 and a connection cable, which can be fitted into theconnector.

Referring now to FIGS. 5A and 5B, an internal configuration of thecontroller 7 will be described. FIG. 5A is a perspective view showingthe controller 7 with an upper casing (a part of the housing 71) takenoff. FIG. 5B is a perspective view showing the controller 7 with a lowercasing (a part of the housing 71) taken off. FIG. 5A shows one side of asubstrate 700, and FIG. 5B shows the other side thereof.

In FIG. 5A, the substrate 700 is secured in the housing 71, and thecontrol buttons 72 a to 72 h, an acceleration sensor 701, the LEDs 702,a quartz oscillator 703, a wireless module 753, an antenna 754, etc.,are provided on the upper principal plane of the substrate 700. Thesecomponents are connected to a microcomputer 751 (see FIG. 6) via lines(not shown) formed on the substrate 700, etc. With the wireless module753 and the antenna 754, the controller 7 can function as a wirelesscontroller.e The quartz oscillator 703 generates a basic clock for themicrocomputer 751.

Referring to FIG. 5B, the image capturing/processing section 74 isprovided at the front edge on the lower principal plane of the substrate700. The image capturing/processing section 74 includes an infraredfilter 741, a lens 742, the image sensing device 743 and an imageprocessing circuit 744 provided in this order from the front side of thecontroller 7, and these components are provided on the lower principalplane of the substrate 700. The connector 73 is provided at the rearedge on the lower principal plane of the substrate 700. The controlbutton 72 i is provided on the lower principal plane of the substrate700 behind the image capturing/processing section 74, and battery cells705 are accommodated in a position further behind the control button 72i. A vibrator 704 is provided on the lower principal plane of thesubstrate 700 between the battery cells 705 and the connector 73. Thevibrator 704 may be, for example, a vibrating motor or a solenoid. Asthe vibrator 704 is actuated, the controller 7 is vibrated, and thevibration is transmitted to the hand of the player holding thecontroller 7, thus realizing a video game with vibration feed back.

Referring now to FIG. 6, an internal configuration of the controller 7will be described. FIG. 6 is a block diagram showing a configuration ofthe controller 7.

Referring to FIG. 6, in addition to the control section 72 and the imagecapturing/processing section 74, the controller 7 includes therein thecommunications section 75 and the acceleration sensor 701.

Specifically, the image capturing/processing section 74 includes theinfrared filter 741, the lens 742, the image sensing device 743 and theimage processing circuit 744. The infrared filter 741 passes only aninfrared portion of incident light entering the controller 7 from thefront side. The lens 742 condenses the infrared light passing throughthe infrared filter 741, and outputs the condensed infrared light to theimage sensing device 743. The image sensing device 743 is a solid-stateimage sensing device, such as a CMOS sensor or a CCD, for capturing theinfrared light condensed through the lens 742. Therefore, the imagesensing device 743 produces image data by capturing only the infraredlight that has passed through the infrared filter 741. The image dataproduced by the image sensing device 743 is processed in the imageprocessing circuit 744. Specifically, the image processing circuit 744processes the image data obtained from the image sensing device 743 todetect high-luminance portions and obtain positions and areas thereof,and the image processing circuit 744 outputs the process result datarepresenting the obtained positions and areas to the communicationssection 75. The image capturing/processing section 74 is secured in thehousing 71 of the controller 7, and the image-capturing direction can bechanged by changing the direction of the housing 71 itself. As willlater be more apparent, it is possible to obtain a signal according tothe position or movement of the controller 7 based on the process resultdata outputted from the image capturing/processing section 74.

The acceleration sensor 701 detects the acceleration in each of threedirections, i.e., the up/down direction, the left/right direction andthe forward/backward direction of the controller 7. The accelerationsensor 701 may alternatively be an acceleration sensor capable ofdetermining the acceleration in each of two directions, i.e., theup/down direction and the left/right direction, depending on types ofcontrol signals necessary. The data representing the accelerationdetermined by the acceleration sensor 701 is outputted to thecommunications section 75.

As a non-limiting example, the three-axis or two-axis linearaccelerometer 701 may be of the type available from Analog Devices, Inc.or STMicroelectronics N.V. Preferably, the acceleration sensor 701 is anelectrostatic capacitance or capacitance-coupling type that is based onsilicon micro-machined MEMS (microelectromechanical systems) technology.However, any other suitable accelerometer technology (e.g.,piezoelectric type or piezoresistance type) now existing or laterdeveloped may be used to provide the three-axis or two- axisacceleration sensor 701.

As one skilled in the art understands, linear accelerometers, as used inacceleration sensor 701, are only capable of detecting accelerationalong a straight line corresponding to each axis of the accelerationsensor. In other words, the direct output of the acceleration sensor 701is limited to signals indicative of linear acceleration (static ordynamic) along each of the two or three axes thereof. As a result, theacceleration sensor 701 cannot directly detect movement along anon-linear (e.g. arcuate) path, rotation, rotational movement, angulardisplacement, tilt, position, attitude or any other physicalcharacteristic.

However, through additional processing of the linear accelerationsignals output from the acceleration sensor 701, additional informationrelating to the controller 7 can be inferred or calculated, as oneskilled in the art will readily understand from the description herein.For example, by detecting static, linear acceleration (i.e., gravity),the linear acceleration output of the acceleration sensor 701 can beused to infer tilt of the object relative to the gravity vector bycorrelating tilt angles with detected linear acceleration. In this way,the acceleration sensor 701 can be used in combination with themicro-computer 751 (or another processor) to determine tilt, attitude orposition of the controller 7. Similarly, various movements and/orpositions of the controller 7 can be calculated or inferred throughprocessing of the linear acceleration signals generated by theacceleration sensor 701 when the controller 7 containing theacceleration sensor 701 is subjected to dynamic accelerations by, forexample, the hand of a user, as explained herein. In another embodiment,the acceleration sensor 701 may include an embedded signal processor orother type of dedicated processor for performing any desired processingof the acceleration signals output from the accelerometers therein priorto outputting signals to micro-computer 751. For example, the embeddedor dedicated processor could convert the detected acceleration signal toa corresponding tilt angle when the acceleration sensor is intended todetect static acceleration (i.e., gravity).

In another exemplary embodiment, the acceleration sensor 701 may bereplaced with a gyro-sensor of any suitable technology incorporating,for example, a rotating or vibrating element. Exemplary MEMSgyro-sensors that may be used in this embodiment are available fromAnalog Devices, Inc. Unlike the linear acceleration sensor 701, agyro-sensor is capable of directly detecting rotation (or angular rate)around an axis defined by the gyroscopic element (or elements) therein.Thus, due to the fundamental differences between a gyro-sensor and anlinear acceleration sensor (e.g., angle-based vs. vector-based output),corresponding changes need to be made to the processing operations thatare performed on the output signals from these devices depending onwhich device is selected for a particular application. Due to the factthat the nature of gyroscopes is known to one skilled in the art, aswell as the fundamental differences between linear accelerometers andgyroscopes, further details are not provided herein so as not to obscurethe remainder of the disclosure. While gyro-sensors provide certainadvantages due to their ability to directly detect rotation, linearacceleration sensors are generally more cost effective when used inconnection with the controller applications described herein.

The communications section 75 includes the microcomputer 751, a memory752, the wireless module 753 and the antenna 745. The microcomputer 751controls the wireless module 753 for wirelessly transmitting transmitdata while using the memory 752 as a memory area.

A control signal (key data) from the control section 72 provided in thecontroller 7, an acceleration signal (acceleration data) from theacceleration sensor 701 and process result data from the imagecapturing/processing section 74 are outputted to the microcomputer 751.The microcomputer 751 temporarily stores the received data (the keydata, the acceleration data and the process result data) in the memory752 as transmit data to be transmitted to the receiver unit 6. Data arewirelessly transmitted from the communications section 75 to thereceiver unit 6 at regular intervals. Since the game process typicallyproceeds in a cycle of 1/60 second, the interval should be shorter than1/60 second. Specifically, the game process proceeds in a cycle of 16.7ms ( 1/60 second), and the data transmission interval of thecommunications section 75 using the Bluetooth (registered trademark)technique is 5 ms. At the transmission timing for transmitting data tothe receiver unit 6, the microcomputer 751 outputs, as a series ofcontrol data, transmit data stored in the memory 752 to the wirelessmodule 753. The wireless module 753 uses a technique such as Bluetooth(registered trademark) to transforms control data into a radio wavesignal using a carrier of a predetermined frequency, and radiates theradio wave signal from the antenna 754. Thus, the key data from thecontrol section 72 provided in the controller 7, the acceleration datafrom the acceleration sensor 701 and the process result data from theimage capturing/processing section 74 are transmitted from thecontroller 7. The radio wave signal is received by the receiver unit 6of the video game device 3, and is demodulated and decoded by the videogame device 3, thereby obtaining the series of control data (the keydata, the acceleration data and the process result data). The CPU 30 ofthe video game device 3 performs the game process based on the obtainedcontrol data and the video game program. Where the communicationssection 75 uses a Bluetooth (registered trademark) technique, thecommunications section 75 can also receive transmit data wirelesslytransmitted from other devices.

As shown in FIG. 7, when playing the game on the video game system 1 byusing the controller 7, the player holds the controller 7 in one hand(the right hand, for example) (see FIGS. 8 and 9). The player holds thecontroller 7 with the front side of the controller 7 (the side forreceiving light to be sensed by the image capturing/processing section74) facing toward the monitor 2. The two markers 8L and 8R are providedaround the display screen of the monitor 2. The markers 8L and 8R outputinfrared light to the front side of the monitor 2, and serve as imagingtargets to be captured by the image capturing/processing section 74.

As the player holds the controller 7 so that the front surface thereoffaces the monitor 2, the image capturing/processing section 74 receivesinfrared light outputted from the two markers 8L and 8R. Then, the imagesensing device 743 captures the incident infrared light via the infraredfilter 741 and the lens 742, and the image processing circuit 744processes the captured image. The image capturing/processing section 74detects the infrared light component outputted from the markers 8L and8R, thereby obtaining the positions of the markers 8L and 8R (theposition of the target image) in the captured image or the areainformation thereof. Specifically, the image processing circuit 744analyzes the image data captured by the image sensing device 743 tofirst exclude, from the area information, images that cannot possibly bethe infrared light from the markers 8L and 8R, and then identifyhigh-luminance points to be the positions of the markers 8L and 8R.Then, the image capturing/processing section 74 obtains positioninformation, e.g., the centroid, of the identified bright spots, andoutputs the obtained position information as the process result data.The position information, being the process result data, may becoordinate values with respect to a predetermined reference point in thecaptured image (e.g., the center or the upper left corner of thecaptured image) being the origin, or may alternatively be a vectorrepresenting the difference between the current bright spot position anda reference point being the bright spot position at a predeterminedpoint in time. Thus, the position information of the target image is aparameter used as the difference with respect to a predeterminedreference point, which is defined in the captured image captured by theimage sensing device 743. As the process result data is transmitted tothe video game device 3, the video game device 3 can obtain, based onthe difference between the position information and the reference, theamount of change in the signal according to the movement, theorientation, the position, etc., of the image capturing/processingsection 74, i.e., the controller 7, with respect to the markers 8L and8R. Specifically, as the controller 7 is moved around, the positions ofthe high-luminance points in the image transmitted from thecommunications section 75 change. Therefore, by making a direction inputor a position input according to the change in the positions of thehigh-luminance points, it is possible to make a direction input or aposition input to a three-dimensional space according to the directionin which the controller 7 is moved. In a game process example to bedescribed later, the image capturing/processing section 74 obtains thecentroid position for each of the target images of the markers 8L and 8Rin the captured image, and outputs the obtained centroid position as theprocess result data.

Thus, the image capturing/processing section 74 of the controller 7captures the image of fixed markers (infrared light from the two markers8L and 8R in the present embodiment), whereby it is possible to make acontrol input according to the movement, the orientation, the position,etc., of the controller 7 by processing data outputted from thecontroller 7 in the game process performed by the video game device 3,thus realizing an intuitive control input, different from those usingcontrol buttons and control keys where the player presses the buttons orthe keys. Since the markers are provided around the display screen ofthe monitor 2, a position with respect to the markers can easily beconverted to the movement, the orientation, the position, etc., of thecontroller 7 with respect to the display screen of the monitor 2. Thus,the process result data based on the movement, the orientation, theposition, etc., of the controller 7 can be used as a control input thatis directly reflected on the display screen of the monitor 2. In thevideo game system 1, by using the captured image obtained by capturingthe markers 8L and 8R, the distance from the markers 8L and 8R to thecontroller 7 can also be used as a control input that is directlyreflected on the display screen of the monitor 2, the details of whichwill be described later.

Referring to FIGS. 8 and 9, how the player holds the controller 7 in onehand will be described. FIG. 8 shows the controller 7 being held in theplayer's right hand, as viewed from the front side of the controller 7.FIG. 9 shows the controller 7 being held in the player's right hand, asviewed from the left side of the controller 7.

As shown in FIGS. 8 and 9, the controller 7 has an overall size suchthat it can be held in a hand of an adult or a child. When the player'sthumb is placed on the upper surface of the controller 7 (e.g., near thecross-shaped key 72 a) and the player's index finger is placed in thedepressed portion of the controller 7 (e.g., near the control button 72i), the light receiving port of the image capturing/processing section74 provided on the front side of the controller 7 is exposed to thefront side of the player. It is understood that the controller 7 can beheld similarly by the player's left hand.

With the controller 7 being held in one hand of the player, the playercan easily operate the control section 72, e.g., the cross-shaped key 72a or the control button 72 i. With the controller 7 being held in onehand of the player, the light receiving port of the imagecapturing/processing section 74 provided on the front side of thecontroller 7 is exposed, whereby infrared light from the two markers 8Land 8R can easily be received through the light receiving port. In otherwords, the player can hold the controller 7 in one hand without blockingany function of the image capturing/processing section 74 of thecontroller 7. Therefore, as the player moves the hand holding thecontroller 7 with respect to the display screen, the controller 7 isfurther provided with a control input function in which the movement ofthe player's hand is directly reflected on the display screen.

As shown in FIG. 10, the markers 8L and 8R each have a viewing angle θ1.The image sensing device 743 has a viewing angle θ2. For example, theviewing angle θ1 of each of the markers 8L and 8R is 34° (half angle),and the viewing angle θ2 of the image sensing device 743 is 41°. Whenthe markers 8L and 8R are both present within the viewing angle θ2 ofthe image sensing device 743 and when the image sensing device 743 ispresent within the viewing angle θ1 of the marker 8L and within theviewing angle θ1 of the marker 8R, the video game device 3 detects theposition of the controller 7 (including the distance from the markers 8Land 8R) by using the position data of the high-luminance points of thetwo markers 8L and 8R.

The details of the game process performed in the video game system 1will now be described. First, important data to be used in the gameprocess will be described with reference to FIG. 11. FIG. 11 showsimportant data to be stored in the main memory 33 of the video gamedevice 3.

Referring to FIG. 11, the main memory 33 stores control information Da,control status information Db, controlled object information Dc, etc. Inaddition to those shown in FIG. 11, the main memory 33 also stores otherdata necessary for the game process, such as player character data(e.g., the image data and the position data of the player character),and game space data (e.g., the terrain data).

The control data Da is a series of control data transmitted from thecontroller 7 as transmit data, and is updated to the latest controldata. The control information Da includes first position data Da1 andsecond position data Da2, corresponding to the process result datadescribed above. The first position data Da1 represents the position(coordinates) of the image of one of the two markers 8L and 8R in thecaptured image captured by the image sensing device 743. The secondposition data Da2 represents the position (coordinates) of the image ofthe other marker in the captured image. For example, the position of theimage of a marker is represented by a set of coordinates in an XYcoordinate system of the captured image.

In addition to the position data (the first position data Da1 and thesecond position data Da2) being an example of the process result dataobtained from the captured image, the control information Da includesthe key data Da3 obtained from the control section 72, the accelerationdata Da4 obtained from the acceleration sensor 701, etc. The receiverunit 6 provided in the video game device 3 receives the controlinformation Da transmitted from the controller 7 at a regular interval(e.g., 5 ms), and the received data are stored in a buffer (not shown)of the receiver unit 6. The stored data is read out, for example, in acycle of one frame ( 1/60 second), being the game process interval, andthe latest information is stored in the main memory 33.

The control status information Db represents the status of thecontroller 7, as determined based on the captured image. The controlstatus information Db is data obtained based on the position and thedirection of the target images (markers) included in the captured image.Specifically, the control status information Db includes direction dataDb1, middle point data Db2, distance data Db3, effective range data Db4,a forward/backward position parameter Db5, etc. The direction data Db1represents the direction from the first position data Da1 to the secondposition data Da2. It is assumed herein that the direction data Db1represents a vector extending from the first position data Da1 to thesecond position data Da2. The middle point data Db2 represents theposition (coordinates) of the middle point between the first positiondata Da1 and the second position data Da2. Where the images of the twomarkers (the markers 8L and 8R) are regarded as one target image, themiddle point data Db2 represents the position of the target image. Thedistance data Db3 represents the distance d from the markers 8L and 8Rto the controller 7, as calculated based on the first position data Da1and the second position data Da2. The effective range data Db4represents an effective range (to be described later), which is definedbased on the distance d in order to obtain parameters to be used in thegame process. The effective range data Db4 includes the length (“Lr”)the end points (“min” “max”) and the extension lengths (“a1” “a2”) ofthe effective range. The forward/backward position parameter Db5 is aparameter (“r”) calculated based on the position of the controller 7(i.e., the distance d) with respect to the effective range, and is usedin the game process as, for example, a parameter representing the depthof the three-dimensional virtual game space.

The controlled object information Dc represents the orientation and theposition, or the pointed position, of the controlled object on thescreen or in the game space. The controlled object as used herein refersto an object displayed on the screen or an object in a virtual gamespace. Where a three-dimensional virtual game space is created, thecontrolled object may be a virtual camera used for displaying thevirtual game space on the screen. The controlled object information Dcincludes orientation data Dc1, object position data Dc2, pointedposition data Dc3 and depth position data Dc4. The orientation data Dc1represents the orientation of the controlled object. The object positiondata Dc2 represents the position of the controlled object in the gamespace or the position thereof on the screen. The pointed position dataDc3 represents the position on the screen of the monitor 2 being pointedat by the controller 7 or the vertical/horizontal position in thethree-dimensional virtual game space, as obtained from the firstposition data Da1 and the second position data Da2. The depth positiondata Dc4 represents the position in the depth direction of thethree-dimensional virtual game space, as obtained from theforward/backward position parameter Db5.

Referring now to FIGS. 12 to 19, the details of the game processperformed by the video game device 3 will be described. FIG. 12 is aflow chart showing the game process performed by the video game device3. FIG. 13 shows, in detail, a subroutine of step 52 in FIG. 12 for theprocess of calculating the control status information, the pointedposition data and the depth position data. FIG. 14 shows, in detail, asubroutine of steps 62 and 65 in FIG. 13 for the process of calculatingthe distance d. FIG. 15 shows, in detaii, a subroutine of step 66 inFIG. 13 for the process of calculating the forward/backward positionparameter and re-defining the effective range. FIG. 16 shows theoperation of calculating the distance d. FIGS. 17A and 17B show theoperation of calculating the forward/backward position parameter r anddefining the effective range according to the subroutine shown in FIG.13. FIGS. 18 and 19 each show the relationship between theforward/backward position parameter r and the depth position data zdefined in the three-dimensional game space. In the flow charts of FIGS.12 to 15, parts of the game process performed based on the gameoperation with the controller 7 are shown, and other parts of the gameprocess that are not directly related to the present invention will notbe shown. In FIGS. 12 to 15, each step performed by the CPU 30 isdenoted by an abbreviation “S” plus the step number.

When the power of the video game device 3 is turned ON, the CPU 30 ofthe video game device 3 executes a boot program stored in a boot ROM(not shown), thus initializing various units such as the main memory 33.The video game program stored in the optical disk 4 is loaded to themain memory 33, and the CPU 30 starts executing the video game program.The flow charts of FIGS. 12 to 15 show the game process performed afterthe completion of the process described above.

Referring to FIG. 12, the CPU 30 obtains control data received from thecontroller 7 (step 51), and proceeds to the next step. The CPU 30 storesthe control data in the main memory 33 as the control information Da. Inaddition to the position data representing the positions of the markers8L and 8R in the captured image (the first position data Da1 and thesecond position data Da2), the control data obtained in step 51 includesdata representing how the control section 72 of the controller 7 isbeing operated (the key data Da3), and data representing theacceleration detected by the acceleration sensor 701 (the accelerationdata Da4). In the illustrated example, the communications section 75transmits the control data to the video game device 3 at a predeterminedtime interval (e.g., 5 ms). It is assumed that the CPU 30 uses thecontrol data every frame. Therefore, the operation loop through steps 51to 57 shown in FIG. 12 is iterated once per frame.

Then, the CPU 30 calculates the control status information, the pointedposition data and the depth position data (step 52), and proceeds to thenext step. In the process of calculating the control status information,the pointed position data and the depth position data, the CPU 30calculates the status of the controller 7 (the movement, the position,the orientation, etc., of the controller 7), based on the stored firstand second position data Da1 and Da2 transmitted from the controller 7.Referring now to FIG. 13, the operation of step 52 will be described indetail.

Referring to FIG. 13, the CPU 30 determines whether or not to performthe initialization process (step 61). For example, where the effectiverange has already been defined, the CPU 30 determines that theinitialization process has been performed (No in step 61), and proceedsto step 64. Where the effective range has not been defined or where thefirst position data Da1 and the second position data Da2 are oncediscontinued and then resumed (i.e., where the markers 8L and 8R oncefall out of the image-capturing range of the image sensing device 743and then come back into the image-capturing range), the CPU 30determines that the initialization process is needed (Yes in step 61),and proceeds to step 62.

In step 62, the CPU 30 calculates the distance d using the firstposition data Da1 and the second position data Da2, and stores thecalculated distance d as the distance data Db3 in the main memory 33.Referring now to FIGS. 14 and 16, the details of the process ofcalculating the distance d will be described.

Referring to FIGS. 14 and 16, the CPU 30 calculates the distance mi (seeFIG. 16) between two points in the captured image (step 81). The twopoints correspond to the images of the markers 8L and 8R, and thepositions (coordinates) thereof are obtained as the first position dataDa1 and the second position data Da2. Therefore, the CPU 30 cancalculate the distance mi using the first position data Da1 and thesecond position data Da2.

Then, the CPU 30 calculates the width w (see FIG. 16) of an image thatcan be captured by the image sensing device 743 at the position of themarkers 8L and 8R (step 82). Specifically, the CPU 30 calculates thewidth w as follows.w=wi×m/miIn the expression, m is the interval between the markers 8L and 8R(e.g., 30 cm), and is a fixed value. Moreover, wi is the width of thecaptured image of the image sensing device 743 for the width w, and isalso a fixed value. Since the interval m and the width wi are both fixedvalues, they are stored in advance in the storage means (not shown) ofthe video game device 3. The interval m may be any suitable valuedepending on the environment in which the markers 8L and 8R areinstalled by the player. The interval between the installed markers 8Land 8R may be inputted by the player as the interval m, in which casethe expression above can similarly be used as described above.

Then, the CPU 30 uses the width w and the viewing angle θ of the imagesensing device 743 to calculate the distance d between the markers 8Land 8R and the image sensing device 743 (the controller 7) (step 83; seeFIG. 16), and exits the subroutine. The CPU 30 calculates the distance dbased on the following relationship.tan (θ/2)=(w/2)/d=w/2dIn the expression, the viewing angle θ is a fixed value, and istherefore stored in advance in the storage means (not shown) in thevideo game device 3.

Referring back to FIG. 13, after calculating the distance d in step 62,the CPU 30 defines an effective range according to the distance d (step63), and proceeds to step 64. As will later be more apparent, theeffective range is defined by the length Lr, the front end position minand the rear end position max. For example, the CPU 30 defines aneffective range so that the distance d obtained in step 62 coincideswith the center of the effective range, the front end position min orthe rear end position max.

In step 64, the CPU 30 calculates the direction data Db1 from the firstposition data Da1 to the second position data Da2. Specifically, the CPU30 refers to the first position data Da1 and the second position dataDa2 to calculate a vector extending from the first position data Da1 tothe second position data Da2. Then, the CPU 30 stores the calculatedvector data in the main memory 33 as the direction data Db1. Based onthe difference between the direction data Db1 and a predeterminedreference direction, it is possible to calculate the rotation about anaxis extending in a direction perpendicular to the image-capturingsurface of the controller 7.

The CPU 30 also calculates the middle point data Db2 representing themiddle point between the first position data Da1 and the second positiondata Da2. Specifically, the CPU 30 refers to the first position data Da1and the second position data Da2 to calculate the position (coordinates)of the middle point. Then, the CPU 30 stores the calculated middle pointposition data in the main memory 33 as the middle point data Db2. In theillustrated example, the middle point data Db2 represents the positionof the target image (the markers 8L and 8R) in the captured image. Basedon the difference between the middle point data Db2 and a predeterminedreference position, it is possible to calculate the change in the imageposition caused by the change in the position of the controller 7.

Then, the CPU 30 uses the first position data Da1 and the secondposition data Da2 to calculate the distance d, and stores the distance din the main memory 33 as the distance data Db3 (step 65). Then, the CPU30 proceeds to the next step. The distance d is calculated in a similarmanner to that described above with reference to FIGS. 14 and 16, andthus will not be further described below.

Then, the CPU 30 calculates the forward/backward position parameter randre-defines the effective range (step 66), and proceeds to the next step.Referring now to FIGS. 15, 17A and 17B, the details of the process ofcalculating the forward/backward position parameter and re-defining theeffective range will be described.

First, the effective range will be described. The effective range asused herein refers to a range defined for the position of the controller7 with respect to a reference. A value based on the position of thecontroller 7 with respect to the effective range is given to the videogame program and used in the game process. In the present embodiment,the distance d from the markers 8L and 8R to the tip of the controller 7is used as the value representing the position of the controller 7.Therefore, the effective range is defined with respect to the distanced, with the lower limit of the effective range being 0 and the upperlimit thereof being 1. A value between 0 to +1 according to the positionof the controller 7 in the effective range is selected as the valueaccording to the position of the controller 7. For example, the width ofthe effective range is determined to be a width across which the playercan reasonably swing the controller 7 held in the player's hand withoutmoving the feet, e.g., about 10 cm to about 40 cm. For example, wherethe width of the effective range is 30 cm and the distance between themarkers 8L and 8R and the initial position of the controller 7 is 1 m 15cm, the effective range is defined to be 1 m to 1 m 30 cm in terms ofthe distance from the markers 8L and 8R. Then, the value 0.5,representing the position at a distance of 1 m 15 cm, is used in thegame process. While the effective range is defined in the illustratedexample so that its center coincides with the position of the controller7, the present invention is not limited thereto, and the effective rangecan be defined in any manner as long as the initial position of thecontroller 7 is included therein.

This will now be described in greater detail with reference to FIGS. 17Aand 17B. First, FIG. 17A shows a case where a player U moves thecontroller 7 forward and backward with respect to the markers 8L and 8R.In FIG. 17A, “d” denotes the distance from the markers 8L and 8R to thecontroller 7, and “d1”, “d2”, “d3”, “d4” and “d5” each denote thedistance from the markers 8L and 8R to the tip of the controller 7. FIG.17B shows a positional relationship between the effective range and thecontroller 7, and shows how to obtain the value (the forward/backwardposition parameter) r to be used in the game process in that position.In the figure, Lr is the width (length) of the effective range, min isthe front end position (the distance from the markers 8L and 8R to theclosest end of the effective range), and max is the rear end position(the distance from the markers 8L and 8R to the farthest end of theeffective range), wherein Lr=max−min. In the present embodiment, theeffective range is defined so that the forward/backward positionparameter r is calculated to be 0 when the controller 7 (strictlyspeaking, the image sensing device 743) is located at the effectiverange front end position min. The forward/backward position parameter ris calculated to be 1 when the controller 7 is located at the effectiverange rear end position max. The forward/backward position parameter ris calculated to be a value between 0 and 1 according to the position ofthe controller 7 when the controller 7 is between the front end positionmin and the rear end position max of the effective range. When theplayer holding the controller 7 moves the controller 7 forward/backwardbeyond the effective range, the effective range is instantly moved sothat the new position of the controller 7 is the front end position minor the rear end position max.

Specifically, where the controller 7 is at a distance d3 and theeffective range is defined to extend from a distance d2 to a distance d4as shown in state A of FIG. 17A, the value r corresponding to thedistance d3 in the effective range where the distance d2 is 0 and thedistance d4 is 1 is calculated to be ⅓ (0.3), for example. The value ris used in the game process. As the controller 7 is moved from theposition at the distance d3 to the position at the distance d2 as shownin state B, the value r is calculated to be 00.33 to 0 according to themovement. As the controller 7 is further moved to the position at adistance d1 as shown in state C, the effective range is moved along withthe movement of the controller 7, and the value r remains to be 0. Then,as the controller 7 is moved from the position at the distance d1 to theposition at the distance d2 as shown in state D, the effective rangeremains unchanged, and the value r according to the position of thecontroller 7 in the effective range is calculated to be ⅓ (0.33), forexample.

Referring to FIG. 15, the process of calculating the forward/backwardposition parameter r and re-defining the effective range will bedescribed. Referring to FIG. 15, the CPU 30 determines whether or notthe distance d calculated in step 65 is shorter than the front endposition min (step 91), and whether or not the distance d is longer thanthe rear end position max (step 92). If the CPU 30 determines that thedistance d is shorter than the front end position min (a transition fromstate B to state C in FIG. 17A), the process proceeds to step 93. If theCPU 30 determines that the distanced is longer than the rear endposition max, the process proceeds to step 94. If the CPU 30 determinesthat the distance d is within the effective range (i.e., if the distanced is equal to or greater than the front end position min and equal to orless than the rear end position max) (state A or state D of FIG. 17A),the process proceeds to step 95 while the effective range remainsunchanged.

In step 93, in response to the movement of the controller 7 to aposition exceeding the effective range in the forward direction, the CPU30 re-defines the effective range, and proceeds to step 95.Specifically, the CPU 30 updates the effective range front end positionmin to the current distance d and the rear end position max to thecurrent distance d plus the length Lr, and stores the updated values inthe main memory 33. In step 94, in response to the movement of thecontroller 7 to a position exceeding the effective range in the backwarddirection, the CPU 30 re-defines the effective range, and proceeds tostep 95. Specifically, the CPU 30 updates the effective range rear endposition max to the current distance d, and the front end position minto the current distance d minus the length Lr, and stores the updatedvalues in the main memory 33. With the provision of steps 93 and 94,when the player moves the controller 7 forward/backward beyond theeffective range, the effective range is instantly moved so that the newposition of the controller 7 is the front end position min or the rearend position max.

In step 95, the CPU 30 calculates the forward/backward positionparameter r according to the distance d and the effective range, andexits the subroutine. Specifically, the CPU 30 calculates (normalizes)the forward/backward position parameter r as follows.r=(d−min)/LrIn step 95, the forward/backward position parameter r is calculated tobe a value between 0 to 1 according to the effective range and theposition of the controller 7.

Referring back to FIG. 13, after the process of calculating theforward/backward position parameter and re-defining the effective rangein step 66, the CPU 30 sets the depth position data z according to theforward/backward position parameter r (step 67), and proceeds to thenext step. The depth position data z is a value for calculating acoordinate in the depth direction in the game space, and is stored inthe main memory 33 as the depth position data Dc4.

In a case where a player character P is moved by increasing/decreasingthe z coordinate value in the camera coordinate system of the virtualcamera in a game using a three-dimensional virtual space as shown inFIGS. 18 and 19, for example, the z coordinate value in the cameracoordinate system is used as the depth position data z. A predeterminedrange is defined for the z coordinate, and the z coordinate value isdetermined so as to correspond to the value of the forward/backwardposition parameter r with respect to the defined range. With the xcoordinate and the y coordinate, in addition to the z coordinate of thecamera coordinate system, the position (coordinates) of the playercharacter P in the world coordinate system is determined based on apredetermined conversion function, and the obtained position(coordinates) is stored in the main memory 33 as-the object positiondata Dc2. The x coordinate and the y coordinate may be predeterminedfixed values, or may be calculated from the control input through aprocess to be described later. In the example shown in FIG. 18, the xcoordinate value and the y coordinate value are independent of themovement in the depth direction. Therefore, when the player character Pis moved in the game space according to the depth position data zwithout changing the x coordinate value and the y coordinate value, theplayer character P moves in a direction parallel to the viewingdirection of a virtual camera C. In the example shown in FIG. 19, the xcoordinate value and the y coordinate value are changed according to thez coordinate value so that the player character P is displayed at thesame position on the screen after a movement in the depth direction.

Where the axis direction of the camera coordinate system is the same asthat of the world coordinate system, the z coordinate value in the worldcoordinate system may be used as the depth position data z throughoutthe entire process. A predetermined range is defined for the zcoordinate, and the z coordinate value is determined so as to correspondto the value of the forward/backward position parameter r with respectto the defined range. Then, the depth position data z can be used as itis as the z component of the object position data Dc2, whereby there isno computational cost for coordinate conversion. Thus, if there is anaxis extending in the movement direction, a coordinate component in theworld coordinate system or the camera coordinate system may be used asthe depth position data z.

If there is no axis extending in the movement direction, a predeterminedparameter for the depth direction may be used as the depth position dataz throughout the entire process. A predetermined range is defined in theworld coordinate system extending in the movement direction includingthe depth direction component, with the depth position data z beingdefined as a parameter representing a position in the range, and theobject position data Dc2 is calculated by a predetermined function.Therefore, in this case, the depth position data z may be theforward/backward position parameter r itself.

Then, the CPU 30 calculates the pointed position data (step 68), andexits the subroutine. The CPU 30 calculates the pointed position dataDc3 by using the direction data Db1 and the middle point data Db2calculated in step 64, and stores the calculated data in the main memory33.

For example, assume a case where the two markers 8L and 8R are installedon the upper surface of the monitor 2 (see FIG. 7), and the playerpoints at the center of the screen of the monitor 2 using the controller7 whose upper surface is facing up (where the center of the screen isbeing at the center of the image captured by the imagecapturing/processing section 74). Then, in the image captured by theimage capturing/processing section 74, the middle point of the targetimage (the middle point between the markers 8L and 8R) does not coincidewith the pointed position (the center of the screen). Specifically, theposition of the target image in the captured image is shifted upward offthe center of the captured image. If a reference position is set so thatthe center of the screen is pointed at when the target image is at sucha position, the position of the target image in the captured image movesin response to the movement of the controller 7 (in the oppositedirection to that of the movement of the controller 7). Therefore, it ispossible to point at a position in the screen corresponding to themovement direction of the controller 7 by performing a process in whichthe pointed position in the screen is moved according to the movement ofthe position of the target image in the captured image. As to thereference position setting, the player may point at a predeterminedposition on the screen so that the position of the target image at thattime is stored while being associated with a reference pointed position.Alternatively, the reference position may be a predetermined position ifthe positional relationship between the target image and the screen isfixed. The pointed position is calculated by using a function forcalculating the coordinates of a position (the pointed position dataDc3) on the screen of the monitor 2 from the middle point data Db2. Thefunction converts the middle point position (coordinates) calculatedfrom a captured image to a position on the screen being pointed at bythe controller 7 (pointed position) at the time when the image iscaptured. With this function, it is possible to calculate a pointedposition on the screen from a middle point position. In order to converta middle point position to a position in the game space, a position onthe screen calculated by the above function can be further converted toa position in the game space corresponding to the position on thescreen. The position in the game space corresponding to the position onthe screen may be the position in the game space that is displayed atthe position on the screen, the three-dimensional coordinate values inthe game space determined by the pointed position data Dc3 and the depthposition data Dc4, etc.

However, if the player points at the center of the screen of the monitor2 using the controller 7 whose upper surface is not facing up (e.g.,facing to the right), the position of the target image in the capturedimage is shifted off the center of the captured image in a directionother than the upward direction (e.g., to the left). Thus, due to a tileof the controller 7, the movement direction of the controller 7 does notcoincide with the movement direction of the pointed position in thescreen. In view of this, the middle point data Db2 is corrected based onthe direction data Db1. Specifically, the middle point data Db2calculated in step 64 is corrected to a middle point position that wouldbe obtained if the upper surface of the controller 7 were facing up.More specifically, a direction data reference is also set in thereference position setting process, and the middle point data Db2calculated in step 64 is corrected by rotating the position representedby the middle point data Db2 about the center of the captured image byan amount according to the angular difference between the direction dataDb1 and the reference direction. Then, the pointed position data Dc3 iscalculated using the corrected middle point data Db2.

An essential principle of the process of calculating the pointedposition data Dc3 is to calculate the displacement of a two-dimensionalposition to be specified with respect to a predetermined referenceposition based on a change in the position of the target image inresponse to the movement of the controller 7. Therefore, the process ofcalculating the pointed position data Dc3 (step 66) can be used not onlyas a coordinate input corresponding to a position in the screen, but canbe used widely as a two-dimensional coordinate input. For example, wherethe depth position data Dc4 is the z coordinate value in the worldcoordinate system as described above with reference to FIGS. 18 and 19,the pointed position data Dc3 can be used as the x coordinate value andthe y coordinate value in the world coordinate system to thereby definea three-dimensional position. The pointed position data calculationprocess in that case may be performed by, irrespective of the screen,performing a calculation process of associating a movement of the targetimage with a movement of the x coordinate and the y coordinate in theworld coordinate system from the reference position. The pointedposition data calculation process does not have to be performed in acase where inputs other than that in the depth direction are notnecessary, e.g., where the x coordinate value and the y coordinate valueare predetermined fixed values.

Referring back to FIG. 12, after the process of calculating the controlstatus information, the pointed position data and the depth positiondata in step 52, the CPU 30 defines a controlled object controlled withthe controller 7 (step 53), and proceeds to the next step. The CPU 30defines a controlled object for each of the control section 72, theimage capturing/processing section 74, the acceleration sensor 701,etc., provided in the controller 7.

For example, a controlled object is updated as necessary in response toadvancements in the game and the player's control operation. Forexample, a controlled object to be controlled based on a signal from theimage capturing/processing section 74 provided in the controller 7,i.e., an object to be controlled based on values of the pointed positiondata Dc3, the depth position data Dc4, etc., may be the object presentat a position in the virtual game world that is obtained by convertingthe pointed position or the depth position. There may be cases whereother objects are controlled at the same time by using control meansother than the signal from the image capturing/processing section 74.For example, the controlled object to be controlled by the cross-shapedkey 72 a provided on the controller 7 may be one of the objects in thevirtual game world that are being displayed on the monitor 2.

In other examples, where it is not necessary to dynamically update acontrolled object, the controlled object may be defined statically. Forexample, a controlled object to be controlled based on a signal from theimage capturing/processing section 74 provided in the controller 7 maybe statically defined to be a player character in the virtual gamespace, the virtual camera for displaying the virtual game space on thescreen, or the like. Where the controlled object is defined statically,it does not have to be defined repeatedly for each iteration of a loop,in which case step 53 may be performed only at the initializationprocess of the game process.

Then, the CPU 30 performs an operation for a controlled object (step54). In step 54, the CPU 30 updates the orientation data Dc1 and theobject position data Dc2 of the controlled object defined in step 53.For example, three-dimensional coordinates are calculated based on thepointed position data Dc3 and the depth position data Dc4, and theobject position data Dc2 is updated with the calculated coordinates.Where the position values except for the depth value are staticallydefined, the three-dimensional coordinates may be calculated from thedefined values and the depth position data Dc4. Where the controlledobject of the image capturing/processing section 74 provided in thecontroller 7 is a player character, an object in the game space or avirtual camera, the player character, the object or the virtual camerais moved to the position represented by the object position data Dc2.The orientation data Dc1 is calculated and updated as necessary inresponse to the movement. Where the controlled object is the virtualcamera, the orientation data Dc1 of the virtual camera may be updated sothat the position represented by the object position data Dc2 is thepoint of sight. Also with objects to be controlled by control meansother than the image capturing/processing section 74, a process isperformed based on a control operation.

Then, the CPU 30 performs a display process (step 56), and proceeds tothe next step. The CPU 30 produces a game image referring to thecontrolled object information Dc and other data necessary for producingthe game image (e.g., data representing the image and the position ofthe player character, and the terrain data) which are stored in the mainmemory 33. Specifically, the CPU 30 produces an image of an object whoseposition and orientation have been updated in step 54, or an image ofthe virtual space as viewed from a virtual camera whose position andorientation have been updated. Then, the CPU 30 displays the producedimage on the screen of the monitor 2.

Then, the CPU 30 determines whether or not to end the game (step 57).For example, the game is ended when a game-over condition is met (e.g. ,when the parameter representing the player character's remainingphysical strength becomes 0) or when the player manually ends the game.If the game is not to be ended, the CPU 30 returns to step 51 to repeatthe process. If the game is to be ended, the CPU 30 exits the processshown in the flow chart.

Thus, by capturing the image of fixedly-installed markers (infraredlight from the two markers 8L and 8R in the illustrated example) withthe image capturing/processing section 74 of the controller 7, it ispossible to realize a novel control device capable of athree-dimensional input operation of specifying not only the position ofthe controller 7 in the up/down direction and the left/right directionwith respect to the markers, but also the distance between the markersand the controller 7. While the range for the input of the distancebetween the markers and the controller 7 may extend over quite a largedistance, an appropriate effective range is defined depending on theplayer's current operating position, thereby allowing for an appropriateinput suitable for the game process while following changes in thedistance. Since the effective range is moved according to the player'scurrent operating position, the player can get a similar operation feelno matter where the player operates the controller 7, thus increasingthe freedom of the position at which the player performs a controloperation. In the present embodiment, when the controller 7 is movedforward/backward beyond the effective range, the effective range ismoved instantly according to the position of the controller 7.Therefore, the present embodiment is suitable for a game process inwhich there should be a high sensitivity to the distance between thecontroller 7 and the markers.

(Second Embodiment)

A video game system according to a second embodiment of the presentinvention will now be described. The video game system of the secondembodiment is similar to that of the first embodiment in terms of thesystem configuration and the operation, except for the process ofcalculating the forward/backward position parameter r and re-definingthe effective range (see step 66 of FIG. 13). Therefore, in thefollowing description of the second embodiment, like elements/steps tothose of the first embodiment will be denoted by like referencenumerals/step numbers, and will not be further described.

Referring to FIGS. 20 to 22B, the process of calculating theforward/backward position parameter and re-defining the effective rangeof the second embodiment will now be described in detail. FIG. 20 shows,in detail, a subroutine for the process of calculating theforward/backward position parameter and re-defining the effective rangeaccording to the present embodiment. FIG. 21 shows, in detail, asubroutine of step 106 in FIG. 20 for the rounding process. FIGS. 22Aand 22B show the operation of calculating the forward/backward positionparameter r and defining the effective range according to the subroutineshown in FIG. 20.

Referring to FIGS. 22A and 22B, the effective range used in the secondembodiment will first be described. As in the first embodiment, theeffective range is a range defined for obtaining the forward/backwardposition parameter r used in the game process in response to a player'soperation in which the player holding the controller 7 moves thecontroller 7 toward or away from the markers 8L and 8R, i.e., anoperation such that the distance d is changed. Therefore, in the processof the present embodiment, the value used as the position of thecontroller 7 is specifically the value of the distance d at that time.The effective range is a range with respect to the distance d, whereinLr is the length of the effective range, min is the front end position(the distance from the markers 8L and 8R to the closest end of theeffective range), and max is the rear end position (the distance fromthe markers 8L and 8R to the farthest end of the effective range) Thus,Lr=max−min.

The effective range is defined so that the forward/backward positionparameter r is calculated to be 0 when the controller 7 (strictlyspeaking, the image sensing device 743) is located at the effectiverange front end position min (e.g., state B of FIG. 22A). Theforward/backward position parameter r is calculated to be 1 when thecontroller 7 is located at the effective range rear end position max.The forward/backward position parameter r is calculated to be a valuebetween 0 and 1 according to the position of the controller 7 when thecontroller 7 is between the front end position min and the rear endposition max of the effective range (e.g., state A of FIG. 22A).

In the present embodiment, a play range having a length of p is providedat both ends of the effective range. The effective range is not movedeven when the controller 7 is moved beyond the effective range if thecontroller 7 is within the play range, which is provided at both ends ofthe effective range to be continuous with the effective range. The playrange is defined so that the forward/backward position parameter r iscalculated to be 0 when the controller 7 is located in the play range atthe front end of the effective range (e.g., state C of FIG. 22A). Theforward/backward position parameter r is calculated to be 1 when thecontroller 7 is located in the play range at the rear end of theeffective range. Thus, when the player holding the controller 7 movesthe controller 7 into a play range at the front or rear end of theeffective range, the forward/backward position parameter r is calculatedto be equal to the front end position min or the rear end position max,whichever is being the boundary between the play range and the effectiverange.

Even though the controller 7 being within a play range means that thecontroller 7 has been moved forward/backward beyond the effective range,the effective range will not be moved as long as the controller 7 islocated within the effective range or the play range (e.g., a transitionfrom state C to state D in FIG. 22A). When the player holding thecontroller 7 moves the controller 7 to a position beyond the play range,the effective range is moved instantly so that the new position of thecontroller 7 is the front end position min or the rear end position max(e.g., a transition from state C to state E in FIG. 22A).

Referring to FIG. 20, the process of calculating the forward/backwardposition parameter r and re-defining the effective range will bedescribed. Referring to FIG. 20, the CPU 30 determines whether or notthe distance d calculated in step 65 (see FIG. 13) is shorter than thefront end position min minus p (step 101; whether or not the controller7 is beyond the end of the front-side play range) and whether or not thedistance d is longer than the rear end position max plus p (step 102;whether or not the controller 7 is beyond the end of the rear-side playrange). If the CPU 30 determines that the distance d is shorter than thefront end position min minus p (a transition from state C to state E inFIG. 22A), the process proceeds to step 103. If the CPU 30 determinesthat the distance d is longer than the rear end position max plus p, theprocess proceeds to step 104. If the CPU 30 determines that the distanced is within the effective range or the play range (i.e., if the distanced is equal to or greater than the front end position min minus p andequal to or less than the rear end position max plus p) (state A tostate D of FIG. 22A), the process proceeds to step 105 while theeffective range remains unchanged.

In step 103, in response to the movement of the controller 7 beyond thefront-side play range, the CPU 30 re-defines the effective range, andproceeds to step 105. Specifically, the CPU 30 updates the effectiverange front end position min to the current distance d and the rear endposition max to the current distance d plus the length Lr, and storesthe updated values in the main memory 33. In step 104, in response tothe movement of the controller 7 beyond the rear-side play range, theCPU 30 re-defines the effective range, and proceeds to step 105.Specifically, the CPU 30 updates the effective range rear end positionmax to the current distance d, and the front end position min to thecurrent distance d minus the length Lr, and stores the updated values inthe main memory 33. With the provision of steps 103 and 104, when theplayer moves the controller 7 forward/backward beyond the play range,the effective range is instantly moved so that the new position of thecontroller 7 is the front end position min or the rear end position max.

In step 105, the CPU 30 calculates the forward/backward positionparameter r according to the distance d and the effective range. Then,the CPU 30 performs a process of rounding the calculated value of theforward/backward position parameter r (step 106), and exits thesubroutine. Specifically, in step 105, the CPU 30 calculates theforward/backward position parameter r as follows.r=(d−min)/LrThen, in the process of step 106, the forward/backward positionparameter r is rounded to a number between 0 and 1 according to theposition of the controller 7 with respect to the effective range and theplay range.

The rounding process will be described with reference to FIG. 21. TheCPU 30 determines whether or not the calculated forward/backwardposition parameter r is smaller than 0 (step 111) and whether or not thecalculated forward/backward position parameter r is larger than 1 (step112). If the forward/backward position parameter r is smaller than 0,the CPU 30 sets the forward/backward position parameter r to 0 (step113), and exits the subroutine. If the forward/backward positionparameter r is larger than 1, the CPU 30 sets the forward/backwardposition parameter r to 1 (step 114), and exits the subroutine. If theforward/backward position parameter r is greater than or equal to 0 andless than or equal to 1, the CPU 30 exits the subroutine withoutchanging the value r.

Thus, also in the second embodiment, while the range for the input ofthe distance between the markers and the controller 7 may extend overquite a large distance, an appropriate effective range is defineddepending on the current operating position, thereby allowing for anappropriate input suitable for the game process while following changesin the distance. Since the effective range is moved according to theplayer's current operating position, the player can get a similaroperation feel no matter where the player operates the controller 7.With the provision of the play range at both ends of the effectiverange, it is possible to realize an operation feel different from thatof the first embodiment. For example, in the first embodiment, when thecontroller 7 is moved forward/backward beyond the effective range, theeffective range is instantly moved according to the new position of thecontroller 7. Thus, the position of the effective range is sensitive tothe distance between the controller 7 and the markers. In the secondembodiment, with the provision of the play range at both ends of theeffective range, there is provided a margin for the movement of theeffective range. Specifically, the player may possibly move thecontroller 7 over a greater distance than intended, which may result inthe controller 7 being moved beyond the effective range even though theplayer does not intend to change the operating position. Even in such acase, the effective range is not moved as long as the controller 7 iswithin the play range in the second embodiment, whereby it is possibleto make a distance input with some margin by using the half-fixedeffective range.

(Third Embodiment)

A video game system according to a third embodiment of the presentinvention will now be described. The video game system of the thirdembodiment is similar to that of the first embodiment in terms of thesystem configuration and the operation, except for the process ofcalculating the forward/backward position parameter r and re-definingthe effective range (see step 66 of FIG. 13). Therefore, in thefollowing description of the third embodiment, like elements/steps tothose of the first embodiment will be denoted by like referencenumerals/step numbers, and will not be further described.

Referring to FIGS. 23 to 24B, the process of calculating theforward/backward position parameter and re-defining the effective rangeof the third embodiment will now be described in detail. FIG. 23 shows,in detail, a subroutine for the process of calculating theforward/backward position parameter and re-defining the effective rangeaccording to the present embodiment. FIGS. 24A and 24B show theoperation of calculating the forward/backward position parameter r anddefining the effective range according to the subroutine shown in FIG.23.

Referring to FIGS. 24A and 24B, the effective range used in the thirdembodiment will first be described. As in the first embodiment, theeffective range is a range defined for obtaining the forward/backwardposition parameter r used in the game process in response to a player'soperation in which the player holding the controller 7 moves thecontroller 7 toward or away from the markers 8L and 8R, i.e., anoperation such that the distance d is changed. Therefore, in the processof the present embodiment, the value used as the position of thecontroller 7 is specifically the value of the distance d at that time,and the effective range is a range with respect to the distance d. Inthe present embodiment, the effective range can be extended by a1 in theforward direction and by a2 in the backward direction. The extensionlengths a1 and a2 are variables, based on which the effective range isextended/shrunk. Where a1=0 and a2=0, the effective range is notextended, whereby the front end position of the un-extended effectiverange (the distance from the markers 8L and 8R to the closest end of theeffective range) is min, and the rear end position thereof (the distancefrom the markers 8L and 8R to the farthest end of the effective range)is max. Specifically, the front end position is defined as min−a1, therear end position as max+a2, and the length Lr of the effective range asmax−min+a1+a2.

The effective range is defined so that the forward/backward positionparameter r is calculated to be 0 when the controller 7 (strictlyspeaking, the image sensing device 743) is located at the un-extendedeffective range front end position min minus a1 (e.g., state B of FIG.24A). The forward/backward position parameter r is calculated to be 1when the controller 7 is located at the un-extended effective range rearend position max plus a2. The forward/backward position parameter r iscalculated to be a value between 0 and 1 according to the position ofthe controller 7 when the controller 7 is between the front end positionmin minus a1 and the rear end position max plus a2 of the un-extendedeffective range (e.g., state A of FIG. 24A).

In the present embodiment, a play range having a length of p is providedat both ends, as shown in FIG. 24B. The effective range is not moved (aswill later be more apparent, the effective range may be extended/shrunk)even when the controller 7 is moved beyond the area between min and maxif the controller 7 is within the play range, which is provided next tomin and max to be continuous with the effective range.

When the controller 7 is located within the play range, the extensionlength a1 or a2 is increased/decreased to extend/shrink the effectiverange so that the boundary of the effective range approaches theposition of the controller 7 at a predetermined rate. When thecontroller 7 is within the play range and is outside the effectiverange, the extension length a1 or a2 is increased to extend theeffective range (e.g., as indicated by an arrow shown in state C of FIG.24A). When the controller 7 is within the play range and within theeffective range, the extension length a1 or a2 is decreased to shrinkthe effective range (e.g., as indicated by an arrow shown in state D ofFIG. 24A). When the controller 7 is located between min and max, theextension length a1 or a2 is decreased to shrink the effective range sothat the boundary of the effective range approaches min and max at apredetermined rate. When the player holding the controller 7 moves thecontroller 7 beyond the play range, the extension length a1 or a2 isequal to the length of the play range, and the entire effective range ismoved so that the boundary of the effective range approaches theposition of the controller 7 at a predetermined rate (e.g., state E ofFIG. 24A). Then, the controller 7 is located outside the effective rangeand outside the play range, and the value of the forward/backwardposition parameter r is adjusted by a process similar to the roundingprocess so that the value is 0 when the controller 7 is on the frontside of the effective range and the play range and the value is 1 whenthe controller 7 is on the rear side of the effective range and the playrange. While one end of the effective range is extending/shrinking, theother end thereof extends/shrinks independently of the first end unlessthe effective range itself is being moved (state A to state D of FIG.24A).

Referring now to FIG. 23, the process of calculating theforward/backward position parameter r and re-defining the effectiverange will be described. Referring to FIG. 23, the CPU 30 determineswhether or not the distance d calculated in step 65 (see FIG. 13) isgreater than or equal to min and less than or equal to max (step 121),whether or not the distance d is less than or equal to min−p (step 122;whether or not the controller 7 is beyond the end of the front-side playrange), whether or not the distance d is longer than min−p and shorterthan min (step 123), and whether or not the distance d is greater thanor equal to max+p (step 124; whether or not the controller 7 is beyondthe end of the rear-side play range). If the distance d is greater thanor equal to min and less than or equal to max (state A of FIG. 24A), theCPU 30 proceeds to step 125. If the distance d is less than or equal tomin−p (state E of FIG. 24A), the CPU 30 proceeds to step 126. If thedistance d is longer than min−p and shorter than min (state C and stateD of FIG. 24A), the CPU 30 proceeds to step 127. If the distance d isgreater than or equal to max+p, the CPU 30 proceeds to step 128. If thedistance d is longer than max+p and shorter than max, the CPU 30proceeds to step 129.

Instep 125, in response to the controller 7 being located between minand max, the CPU 30 re-defines the effective range by shrinking it toits original length, and proceeds to step 130. Specifically, the CPU 30updates the extension length a1 to a1×s (s is a constant satisfying0<s<=1; the effective range instantly goes back to the original lengthif s=1) and the extension length a2 to a2×s, and stores the updatedvalues in the main memory 33. In step 126, in response to the movementof the controller 7 beyond the front-side play range, the CPU 30re-defines the effective range, and proceeds to step 130. Specifically,the CPU 30 updates the extension length a1 to the length p of the playrange, the extension length a2 to a2×s, the value min tomin−(min−d−a1)×s, and the value max to max−(min−d−a1)×s, and stores theupdated values in the main memory 33. In step 127, in response to themovement of the controller 7 into the front-side play range, the CPU 30re-defines the effective range, and proceeds to step 130. Specifically,the CPU 30 updates the extension length a1 to a1+(min−d−a1)×s and theextension length a2 to a2×s, and stores the updated values in the mainmemory 33. In step 128, in response to the movement of the controller 7beyond the rear-side play range, the CPU 30 re-defines the effectiverange, and proceeds to step 130. Specifically, the CPU 30 updates theextension length a1 to a1×s, the extension length a2 to the length p ofthe play range, the value max to max+(d−max−a2)×s and the value min tomin+(d−max−a2)×s, and stores the updated values in the main memory 33.In step 129, in response to the movement of the controller 7 into therear-side play range, the CPU 30 re-defines the effective range, andproceeds to step 130. Specifically, the CPU 30 updates the extensionlength a1 to a1×s and the extension length a2 to a2+(d−max−a2)×s, andstores the updated values in the main memory 33. With the provision ofsteps 125 to 129, the effective range is extended/shrunk and moved at apredetermined speed according to the new position of the controller 7.

In step 130, the CPU 30 calculates the forward/backward positionparameter r according to the distance d and the effective range. Then,the CPU 30 performs a process of rounding the calculated value of theforward/backward position parameter r (step 131), and exits thesubroutine. Specifically, in step 130, the CPU 30 calculates the lengthLr as followsLr=max−min+a1+a2and calculates the forward/backward position parameter r as follows.r=(d−min−a1)/LrIn step 131, the forward/backward position parameter r is rounded to anumber between 0 and 1 according to the position of the controller 7with respect to the effective range and the play range. The roundingprocess of step 131 is similar to that of the second embodiment asdescribed above with reference to FIG. 21, and will not be furtherdescribed below.

Thus, also in the third embodiment, while the range for the input of thedistance between the markers and the controller 7 may extend over quitea large distance, an appropriate effective range is defined depending onthe current operating position, thereby allowing for an appropriateinput suitable for the game process while following changes in thedistance. Since the effective range is moved according to the player'scurrent operating position, the player can get a similar operation feelno matter where the player operates the controller 7. With the provisionof the play range at both ends of the effective range, and with theeffective range being extended/shrunk at a predetermined rate and movedat a predetermined rate, it is possible to realize an operation feeldifferent from that of the first or second embodiment. For example, inthe first and second embodiments, when the controller 7 is movedforward/backward beyond the effective range or beyond the play range,the effective range is instantly moved according to the new position ofthe controller 7. Thus, the position of the effective range is sensitiveto the distance between the controller 7 and the markers. In the thirdembodiment, with the provision of the play range at both ends of theeffective range and with the effective range being extended/shrunk ormoved gradually, the effective range reacts even more slowly to aplayer's input. Even if the player unintentionally makes an abrupt,erroneous movement, thereby moving the controller 7 forward/backwardbeyond the effective range, the effective range will be following theoperating position only gradually. Therefore, the influence of theerroneous operation can be absorbed to some extent. If the playerchanges the position of the controller 7 intending to move the effectiverange, the effective range is moved appropriately. Thus, it is possibleto realize a more stable input operation.

While the effective range is defined so that it is possible to make aninput specifying the distance between the markers and the controller 7by capturing the image of fixedly-installed markers with the imagecapturing/processing section 74 of the controller 7, it mayalternatively be defined so that a position in other directions can bespecified based on the captured image. For example, in order to specifythe position of the controller 7 with respect to the markers in theleft/right direction or that in the up/down direction (e.g., the pointedposition data Dc3), an effective range may be defined to extend in eachof these directions. Then, a left/right or up/down position parametercan be calculated based on the positional relationship between theposition of the controller 7 in the left/right direction or the up/downdirection and the effective range. It is possible to define an effectiverange for making an input specifying a position in other directions bymoving or extending/shrinking the effective range according to theposition of the controller 7 in the left/right direction or the up/downdirection, as with the distance direction as described above.

This variation can be realized in the form of a video game program asfollows. The video game program is a video game program executed by acomputer (30) of a video game device (3) for performing a game processby obtaining, as control data (Da), data (Da1, Da2) from a gamecontroller (7) including an image-capturing section (74) for capturingan image of an imaging target (8), the data (Da1, Da2) being either thecaptured image itself from the image-capturing section or data obtainedby performing a predetermined operation on the captured image. The videogame program instructs the computer to perform a position detectionstep, an effective range defining step, a position data calculationstep, a game process step and a display process step. The positiondetection step uses the control data obtained from the game controllerto detect at least one of the position of the game controller withrespect to the imaging target in the up/down direction and that in theleft/right direction, based on the position of the imaging target in thecaptured image. The effective range defining step defines an effectiverange having a predetermined length along at least one of the up/downdirection and the left/right direction with respect to the gamecontroller. The position data calculation step calculates position dataregarding the position in at least one of the up/down direction and theleft/right direction, according to the position of the game controllerwith respect to the effective range. The game process step performs apredetermined game process on the virtual game world based on theposition data calculated in the position data calculation step. Thedisplay process step displays, on a display device, an image of thevirtual game world after the game process. The video game program mayinstruct the computer to further perform an effective range re-definingstep. When the position of the game controller goes beyond the effectiverange, the effective range re-defining step re-defines the effectiverange by moving the effective range according to the new position of thegame controller.

While the forward/backward position parameter r is converted to thedepth position data z in the game process described above, theforward/backward position parameter r may be used in various types ofgame processes. In an alternative game process, the forward/backwardposition parameter r may be converted to the speed of an object in thevirtual game world, such as a vehicle. In another game process, theforward/backward position parameter r may be used as a parameter for anaction (e.g., throwing a punch) executed by a player character in thevirtual game world. In another game process, the forward/backwardposition parameter r may be converted to the zoom factor of the virtualcamera. In another game process, at least one threshold value may be setfor the forward/backward position parameter r so as to provide a switchthat is turned ON/OFF based on the value r with respect to the thresholdvalue (e.g., a predetermined action of a player character may betriggered based on the value r with respect to the threshold value).

While the distance d is calculated by analyzing the image data obtainedby the image sensing device 743 in the above description, the presentinvention can be carried out with any other suitable means as long asthe distance d to a predetermined measured target placed in a real spacecan be determined, whereby an effective range can be defined for thedistance d. The means for determining the distance d may be, forexample, those using an ultrasonic sensor, a magnetic sensor, etc. It ispossible to make a control input, without determining the actualdistance d between the image sensing device 743 and the imaging target,as long as it is possible to obtain a value associated with thedistance. Therefore, without determining the distance d, the process canbe performed by using data provided in advance and associated with thelength between the markers 8L and 8R in the captured image.

While the controller 7 and the video game device 3 are connected to eachother via wireless communications in the above description, thecontroller 7 and the video game device 3 may be electrically connectedto each other via a cable. In such a case, a cable connected to thecontroller 7 is connected to a connection terminal of the video gamedevice 3.

While the receiver unit 6 connected to the connection terminal of thevideo game device 3 is used as receiver means for receiving transmitdata wirelessly transmitted from the controller 7 in the abovedescription, the receiver means may alternatively be a receiver moduleprovided inside the video game device 3. In such a case, the transmitdata received by the receiver module is outputted to the CPU 30 via apredetermined bus.

In the above description, the image data captured by the image sensingdevice 743 is analyzed, whereby the position of infrared light from themarkers 8L and 8R, the centroid thereof, etc., are produced in thecontroller 7 as process result data, and the produced process resultdata is transmitted to the video game device 3. Alternatively, data atany other suitable process step may be transmitted from the controller 7to the video game device 3. For example, the image data captured by theimage sensing device 743 may be transmitted from the controller 7 to thevideo game device 3, wherein the CPU 30 performs the analysis process toobtain the process result data. In such a case, there is no need for theimage processing circuit 744 provided in the controller 7.Alternatively, data at a certain point during the process of analyzingthe image data may be transmitted from the controller 7 to the videogame device 3. For example, data obtained from the image datarepresenting luminance, position, area, etc., may be transmitted fromthe controller 7 to the video game device 3, wherein the CPU 30 performsthe rest of the analysis process to obtain the process result data.

While infrared light from the two markers 8L and 8R are used as imagingtargets to be captured by the image capturing/processing section 74 ofthe controller 7 in the above description, any other suitable object maybe used as the imaging target. For example, only one marker or three ormore markers may be provided around the monitor 2, and the infraredlight from these markers may be used as imaging targets to be capturedby the image capturing/processing section 74. For example, one markerhaving a predetermined length may be provided around the monitor 2,wherein the present invention can be carried out as described above byusing the length of the marker as the interval m (see FIG. 16).Alternatively, the display screen of the monitor 2 itself or otherlight-emitting objects (e.g., lighting in the room) may be used as theimaging target to be captured by the image capturing/processing section74. Any of various light-emitting objects may be used as the imagingtarget to be captured by the image capturing/processing section 74, bycalculating the position of the controller 7 with respect to the displayscreen based on the positional relationship between the imaging targetand the display screen of the monitor 2.

The shape of the controller 7, and the shape, number and arrangement ofthe control sections 72 provided on the controller 7, are allillustrative, and it is understood that the present invention can becarried out with any other suitable shape, number and arrangement. Theposition of the image capturing/processing section 74 in the controller7 (the light receiving port of the image capturing/processing section74) does not have to be the front side of the housing 71, but may be onany other side as long as light can be received from outside the housing71.

While the distance d is used as the value representing the position ofthe controller 7 in the real space in the first to third embodiments,the three-dimensional position in the real space where the controller 7may be detected, as well as the distance d, and the three-dimensionalposition may be used in the process of determining the position of thecontrolled object.

In such a case, for example, a process similar to that of the first tothird embodiments may be performed for effective ranges defined for thevarious components of the three-dimensional coordinate system, therebydetermining the three-dimensional coordinates in the virtual space.

Alternatively, two predetermined three-dimensional positions may be setin a three-dimensional coordinate system as two boundaries, and theeffective range may be defined as an area extending between the twoboundaries, thereby performing a process based on the position of thecontroller 7 in the three-dimensional area. For example, the effectiverange may be a rectangular parallelepiped defined with the first pointand the second point being opposing apexes. In such a case, each edge ofthe rectangular parallelepiped may be parallel to an axis of thecoordinate system. A rectangular parallelepiped model may be provided inadvance, whereby the effective range can be defined only by adjustingthe size of the rectangular parallelepiped model so that the first pointand the second point are included therein.

The position of the controller 7 in the real space may be calculated asfollows, for example. If the orientation of the controller 7 can beknown based on the acceleration data Da4, it is possible to calculatethe three-dimensional position with respect to the positions of themarkers 8R and 8L based on the first position data Da1 and the secondposition data Da2.

The position of the controller 7 in the real space may be calculated by,for example, measuring the position by using an ultrasonic sensor, amagnetic sensor, etc.

The target to be moved in the real space for making the control inputdoes not have to be the controller 7 including the imagecapturing/processing section 74. For example, a data input deviceincluding an image capturing device may be fixedly installed at apredetermined position, and the position of a target that does notoutput data, e.g., a hand of the player, may be detected by the imagecapturing device, thereby outputting the position data from the datainput device.

The storage medium storing a virtual position determining program of thepresent invention is useful, for example, as a video game system and avideo game program of a video game in which a player holds and operatesa physically-separate controller unit, and by which it is possible torealize a control operation with a high degree of freedom.

While the invention has been described in detail, the foregoingdescription is in all aspects illustrative and not restrictive. It isunderstood that numerous other modifications and variations can bedevised without departing from the scope of the invention.

1. A storage medium storing a virtual position determining program forinstructing a computer to perform a process on a virtual space based oninput information obtained from an input device, the input informationincluding target data that changes according to a change in a positionof a predetermined target in a real space, the program instructing thecomputer to perform: an actual position data calculation step ofcalculating actual position data representing a position of the targetin the real space by using the target data; an effective range definingstep of defining an effective range as a range extending from a firstboundary to a second boundary, wherein the first boundary is representedby first position data and the second boundary is represented by secondposition data; an effective range position data calculation step ofcalculating effective range position data representing a position of avalue of the actual position data in the effective range; a positiondetermining step of determining a position of a controlled object in thevirtual space through an operation based on a value of the effectiverange position data; and a display process step of displaying thevirtual space on a display device.
 2. The storage medium storing avirtual position determining program according to claim 1, wherein thevirtual position determining program instructs the computer to furtherperform an effective range updating step of, when the value of theactual position data is outside the effective range, updating theeffective range by moving the first boundary and the second boundary sothat the value of the actual position data is included in the effectiverange.
 3. The storage medium storing a virtual position determiningprogram according to claim 1, wherein the virtual position determiningprogram instructs the computer to further perform an effective rangeupdating step of, when the value of the actual position data is outsidethe effective range, updating the effective range by moving the firstboundary and the second boundary so that one of the first boundary andthe second boundary that is closer to the value of the actual positiondata becomes equal to the value of the actual position data.
 4. Thestorage medium storing a virtual position determining program accordingto claim 1, wherein the virtual position determining program instructsthe computer to further perform a second effective range updating stepof, when the value of the actual position data is outside the effectiverange, updating the effective range by moving the first boundary and thesecond boundary so that one of the first boundary and the secondboundary that is closer to the value of the actual position dataapproaches the value of the actual position data at a predeterminedrate.
 5. The storage medium storing a virtual position determiningprogram according to claim 1, wherein the effective range position datacalculation step calculates the effective range position data so thatthe effective range position data takes a maximum value when the valueof the actual position data is equal to the first boundary and theeffective range position data takes a minimum value when the value ofthe actual position data is equal to the second boundary.
 6. The storagemedium storing a virtual position determining program according to claim1, wherein: the effective range defining step includes a step of furtherdefining a play range having a predetermined length at a front end ofthe effective range and a rear end thereof; and the virtual positiondetermining program instructs the computer to further perform aneffective range updating step of, when the actual position data takes avalue outside a combined range including the effective range and theplay ranges, updating the effective range by moving the values of thefirst boundary and the second boundary so that one of the first boundaryand the second boundary that is closer to the value of the actualposition data becomes equal to the value of the actual position data. 7.The storage medium storing a virtual position determining programaccording to claim 6, wherein the effective range position datacalculation step includes a step of calculating the effective rangeposition data so that the effective range position data takes a maximumvalue when the value of the actual position data is equal to the firstposition data or exceeds the effective range in a direction of the firstposition data, and the effective range position data takes a minimumvalue when the value of the actual position data is equal to the secondposition data or exceeds the effective range in a direction of thesecond position data.
 8. The storage medium storing a virtual positiondetermining program according to claim 1, wherein: the effective rangedefining step defines the second position data on a first side of thefirst position data, third position data on a second side of the firstposition data, and fourth position data on the first side of the secondposition data; and the effective range defining step defines an upperlimit boundary as position data variable from the first position data tothe third position data and a lower limit boundary as position datavariable from the second position data to the fourth position data, anddefining the effective range as a range between the variable upper limitboundary and the variable lower limit boundary; and the virtual positiondetermining program instructs the computer to further perform: a stepof, when the value of the actual position data is within the variablerange of the upper limit boundary/lower limit boundary, updating theupper limit boundary/lower limit boundary so that the upper limitboundary/lower limit boundary approaches the value of the actualposition data at a predetermined rate, and updating the lower limitboundary/upper limit boundary so that the lower limit boundary/upperlimit boundary approaches the second position data/first position dataat a predetermined rate; a step of, when the value of the actualposition data is within a range between the first position data and thesecond position data, updating the upper limit boundary/lower limitboundary so that the upper limit boundary/lower limit boundaryapproaches the first position data/second position data at apredetermined rate; a step of, when the value of the actual positiondata is further on the second side of the third position data, movingthe first position data, the second position data, the third positiondata, the fourth position data and the lower limit boundary so that avalue of the third position data approaches the value of the actualposition data at a predetermined rate and updating the upper limitboundary to the new second position data; and a step of, when the valueof the actual position data is further on the first side of the fourthposition data, moving the first position data, the second position data,the third position data, the fourth position data and the upper limitboundary so that a value of the fourth position data approaches thevalue of the actual position data at a predetermined rate and updatingthe lower limit boundary to the new fourth position data.
 9. The storagemedium storing a virtual position determining program according to claim8, wherein the effective range position data calculation step includes astep of calculating the effective range position data so that theeffective range position data takes a maximum value when the value ofthe actual position data is equal to the first position data or exceedsthe effective range in a direction of the first position data, and theeffective range position data takes a minimum value when the value ofthe actual position data is equal to the second position data or exceedsthe effective range in a direction of the second position data.
 10. Astorage medium storing a virtual position determining program for, basedon data from an input device including position information of a targetin a real space, calculating a position of the target obtained whenrepresenting the real space in a predetermined coordinate system, andcalculating a virtual position of a virtual object in a virtual spacebased on the calculated position, the virtual position determiningprogram instructs a computer to function as: effective range definingmeans for defining an effective range, which is defined by at least afirst position and a second position in a predetermined coordinatesystem of the real space; normalization means for, when the position ofthe target is included in the effective range, normalizing the effectiverange and calculating a value corresponding to the position of thetarget obtained when the normalization is performed; and calculationmeans for calculating the virtual position based on the value calculatedby the normalization means.
 11. The storage medium storing a virtualposition determining program according to claim 10, wherein the virtualposition determining program instructs the computer to further functionas effective range moving means for, when the position of the target isnot included in the effective range, updating the first position and thesecond position to move the effective range to a position that includesthe position of the target.